1 Acknowledgements

The work on the Statistical Performance Indicators (SPI) has benefited greatly from the advice, comments and input of many individuals outside the World Bank SPI Team. The work took place under the overall guidance of Haishan Fu, Director of the Development Data Group at the World Bank, who played a key role in helping shape the framework of statistical performance developed here. We are grateful to the SPI Technical Advisory Group consisting of former National Statistician of the Philippines and Director of the Philippine Statistics Authority Lisa Bersales, the Director General of the National Institute of Statistics of Rwanda Yusuf Murangwa, the Chief Statistician of Colombia Juan Oviedo, and the former Director General of Eurostat and Chief Statistician of the European Union Walter Radermacher, and chaired by former National Statistician of the UK, John Pullinger.

Several World Bank colleagues provided valuable feedback to this project. They include Claudia P Rodriguez Alas, Rabah Arezki, Joao Pedro Wagner De Azevedo, Benu Bidani, Carlos Rodriguez Castelan, Ximena Del Carpio, Olivier Dupriez, Uche Ekhator, Tony Henri Mathias Jany Fujs, Nada Hamadeh, Craig Hammer, Tim Herzog, Kristen Himelein, Johannes Hoogeveen, Asif Mohammed Islam, Dean Jolliffe, Talip Kilic, Christoph Lakner, Maria Ana Lugo, Daniel Gerszon Mahler, Johan A. Mistiaen, Rinku Murgai, Ambar Narayan, Minh Nguyen, Utz Johann Pape, Martin Rama, Pierella Paci, Carolina Sanchez-Paramo, Ana Florina Pirlea, Emi Suzuki, Emil Daniel Tesliuc, Satoshi Tozaki, Tara Vishwanath, Divyanshi Wadhwa, Matthew Welch, Phillip Wollburg, Salman Zaidi, and Hassan Zaman. Elysee Kiti and Tigist Shibru provided superb administration support.

Additionally, we benefited from discussions with the following: Louis Marc Ducharme, Jimmy McHugh, Brett Humburg, Francis Cicero de Leon Bobadilla, and Aubriana Wolferts (IMF)); Pietro Gennari (FAO); Francesca Perucci and Yongyi Min (UN Statistics Division); Ramesha Saligrama Krishnamurthy and Rifat Hossain (WHO); Viveka Palm (Eurostat); Rafael Diez de Medina, Ritash Sarna, Marie-Claire Sodergren, and Lara Badre (ILO); Silvia Montoya and Adolfo Imhof (UNESCO Institute of Statistics); Paul Schreyer (OECD); Emily Poskett (ONS, UK); Phil Cockerill (FCDO, UK); Johannes Jutting, Francois Fonteneau, Julia Schmidt, Yu Tian, and Archita Misra (PARIS21); Shaida Badiee, Eric Swanson and Jamison Crowell (Open Data Watch); Claire Melamed (GPSDD); Misha Belkindas (ODW Consulting); Mike Hughes (Full Fact); Peter Rosendorff (New York University); James Hollyer (University of Minnesota); and James Vreeland (Princeton University).

We apologize for any omissions and express our sincere gratitude to everyone, whether or not they are named here, who graciously gave their time and expertise. We are grateful to the United Kingdom Foreign, Commonwealth and Development Office (FCDO, formerly named the Department for International Development) for funding assistance through Knowledge for Change (KCP) grants, including one for the World Development Report 2021 “Data for Better Lives.”

2 Motivation

National statistical systems are at the heart of the successful governance of nations. They provide an essential public service, helping governments make decisions about the economy, society and environment and enabling citizens to hold those governments to account.

The 2030 agenda for sustainable development has created a framework of goals and targets of universal applicability, designed to ensure that no one is left behind. This agenda requires national statistical systems to work together with other stakeholders within and across nations and regions to measure progress in a consistent and relevant way. At the same time, the data revolution has resulted in previously unimaginable sources of data becoming available that can be mobilized in support of the 2030 agenda in an open, transparent, and ethical manner.

However, many nations are struggling to build the necessary financial, human and technological capability to meet these goals. Assessing and improving the capacity of national statistical systems has long been part of the global agenda for statistics. Capacity assessment tools have been developed by organizations including PARIS21, the Food and Agriculture Organization of the United Nations (FAO), the United Nations Economic Commission for Europe (UNECE), the United Nations Economic Commission for Africa (UNECA) and the US Census Bureau.

Since 2004, the World Bank’s Statistical Capacity Index (SCI) has been part of this global toolkit. The SCI has been used by several national and international agencies to measure progress in statistical capacity building and related investments. To remain useful in the current and future data landscape, the SCI needs to be updated. The SCI incorporates only a limited number of indicators and does not assess use of data sources beyond traditional censuses and surveys. Neither does the SCI consider data dissemination and how data are used. Finally, the SCI places a large weight on statistical output and activities, while neglecting the infrastructure and resource components of statistical systems.

The COVID-19 pandemic has added extra motivation and impetus for understanding and improving the performance of national statistical systems. Without good statistics, countries are hamstrung in their response to the pandemic and in navigating towards a better future. The statistical performance indicators proposed in this paper are a response to that motivation. The indicators are intended to be applicable to all nations and to be forward looking and comprehensive. They are designed to be used by national governments and statistical offices as well as international agencies and donors. The aim is to help create a learning data ecosystem that can develop and adapt iteratively to the ever-changing requirements of government and citizens for better data to support better decisions.

3 Overview of the SPI Framework

The new Statistical Performance Indicators (SPI) build on the Statistical Capacity Indicators The new Statistical Performance Indicators (SPI) build on the Statistical Capacity Index (SCI), which the World Bank regularly published between 2004 and 2021. The SPI framework covers several of the same attributes as the SCI, such as statistical methodology, data, and periodicity, but expands into new areas as well. The goals of introducing the SPI are: to offer a forward looking framework, to measure all statistical systems – from less mature to highly advanced, to cover the entire national statistical system - not just the National Statistical Office (NSO), and to provide countries incentives to build modern statistical systems. To help build transparency and confidence in the work, as well as to improve understanding and encourage further experimentation, refinement, and improvement of the measures, the data and the code underlying the construction of the SPI are made open.1

Figure 2.1: Virtuous Data Cycle

By helping countries identify the strengths and weaknesses of their national statistical system, the SPI can be a guide to decisions about priorities for investment and can help identify partner countries from which they might learn. This in turn can help both donor and recipient countries ensure that development assistance is targeted on areas of most national benefit. The SPI follows an approach that considers the system as a whole, which incorporates users, producers and partners at various levels. The indicators are designed to be dynamic and forward looking. Improvements in performance can be represented as a virtuous data cycle that can become self-sustaining. In this model, adapted from work done by the Organization for Economic Cooperation and Development, PARIS21 and Open Data Watch, investment in data leads to higher levels of effectiveness which drives innovation, value added and impact.

The impact sought from national statistical systems is better decisions and stronger accountability. The way in which national statistical systems contribute to better decisions and stronger accountability is by meeting user needs for statistics. Such user needs are in turn met by the design and production of statistics and statistical indicators. The process of design and production relies on a wide range of organizations, working in partnership with the national statistical office as part of the national statistical system to develop and utilize richer data sources. To be successful the whole data ecosystem needs to build its capability to adapt and thrive.

Statistics have no value unless they are used. Consequently, the first pillar of the SPI is data use. A successful statistical system is one that produces data products that are highly used. To meet user needs, the statistical system must develop a range of services that connect data users and producers and facilitate dialogue between them. The second pillar of the SPI is therefore data services that are trusted by users. A successful statistical system is one with highly valued and well used statistical services.

The dialogue between users and suppliers in turn drives the design and quality of statistical products that are created to meet the country requirement. These products should incorporate accuracy, timeliness, frequency, comparability, and levels of disaggregation. The third pillar of the SPI is therefore data products. A successful statistical system is one that generates high quality statistical indicators that can also track progress toward the Sustainable Development Goals (SDGs).

In order to create the required products, the statistical system needs to make use of a variety of sources from both within and outside the government. This will include making use of traditional data collection methods like censuses and surveys, and also administrative data, geospatial data, and data generated from the private sector and from citizens. It follows that the fourth pillar of the SPI is data sources. A successful statistical system is one which draws on all types of data sources relevant to the indicators that are to be produced.

To complete the virtuous data cycle, the capability of the statistical system needs to be reviewed continuously to ensure that it is strong enough to deliver the data products and services and to promote data use. The fifth pillar of the SPI is therefore data infrastructure. A successful statistical system is one that develops both hard infrastructure (legislation, governance, standards) and soft infrastructure (skills, partnerships) and has the financial resources to deliver.

4 Pillars and Dimensions of the SPI

The pillars of the SPI establish the essential interconnecting elements of a national statistical system. Each one must be healthy for the system as a whole to generate high levels of value. The pillars correspond to the components of an efficient organizational scorecard: outcomes (benefits for users) are delivered through outputs (services) which are generated by effective internal processes (design and delivery of products). These require the right inputs (data sources) and capabilities (infrastructure).

Each of the five pillars has specific attributes which we call dimensions. These dimensions offer distinct components of the pillars by which to judge a country’s statistical performance. The framework offers 5 pillars and 22 associated dimensions. The framework of the SPI is shown in Figure 3.1 below.

Figure 3.1: The Pillars and Dimensions that Construct the New SPI

The dimensions break down each pillar into the core components or attributes of the system. It is likely that the balance between these dimensions will vary across countries and overtime but all are important, and a mature statistical system will score well against each of them. By setting them out in this way, it is possible for countries to ask questions about whether they have the balance right, given their domestic circumstances, when they compare themselves to other countries. The results provide a tool to explore what future investment is needed and where suitable comparators might be found.

4.1 Pillar 1: Data use

The data use (outcome) pillar is segmented by five types of users: (i) the legislature, (ii) the executive branch, (iii) civil society (including sub-national actors), (iv) academia and (v) international bodies. Each dimension would have associated indicators to measure performance. A mature system would score well across all dimensions whereas a less mature one would have weaker scores along certain dimensions. The gaps would give insights into prioritization among user groups and help answer questions as to why the existing services are not resulting in higher use of national statistics in a particular segment.

4.2 Pillar 2: Data services

The data services (output) pillar is segmented by four service types: (i) the quality of data releases, (ii) the richness and openness of online access, (iii) the effectiveness of advisory and analytical services related to statistics, and (iv) the availability and use of data access services such as secure microdata access. Advisory and analytical services might incorporate elements related to data stewardship services including input to national data strategies, advice on data ethics and calling out misuse of data in accordance with the Fundamental Principles of Official Statistics.

4.3 Pillar 3: Data products

The data products (internal process) pillar is segmented by four topics and organized into (i) social, (ii) economic, (iii) environmental, and (iv) institutional dimensions using the typology of the Sustainable Development Goals (SDGs). This approach anchors the national statistical system’s performance around the essential data required to support the achievement of the 2030 global goals, and enables comparisons across countries so that a global view can be generated while enabling country specific emphasis to reflect the user needs of that country.

4.4 Pillar 4: Data sources

The data sources (input) pillar is segmented by four types of sources generated by (i) the statistical office (censuses and surveys), and sources accessed from elsewhere such as (ii) administrative data, (iii) geospatial data, and (iv) private sector data and citizen generated data. The appropriate balance between these source types will vary depending on a country’s institutional setting and the maturity of its statistical system. High scores should reflect the extent to which the sources being utilized enable the necessary statistical indicators to be generated. For example, a low score on environment statistics (in the data production pillar) may reflect a lack of use of (and low score for) geospatial data (in the data sources pillar). This type of linkage is inherent in the data cycle approach and can help highlight areas for investment required if country needs are to be met.

4.5 Pillar 5: Data infrastructure

The data infrastructure (capability) pillar includes hard and soft infrastructure segments, itemizing essential cross cutting requirements for an effective statistical system. The segments are: (i) legislation and governance covering the existence of laws and a functioning institutional framework for the statistical system; (ii) standards and methods addressing compliance with recognized frameworks and concepts; (iii) skills including level of skills within the statistical system and among users (statistical literacy); (iv) partnerships reflecting the need for the statistical system to be inclusive and coherent; and (v) finance mobilized both domestically and from donors.

4.6 A Dashboard

A score for each indicator, dimension, and pillar, and an overall score derived by aggregating the indicators (over these dimensions or over time), would:

  1. facilitate an understanding of the maturity of the national statistical system (in relation to others, e.g., quintile groups of countries).

  2. highlight relative strengths and weaknesses of the system and give an indication of the extent to which the official statistics can be relied upon.

  3. point to which other countries a particular country could learn from as it seeks to improve and create incentives to develop in a forward looking way.

  4. allow assessments of progress and provide a starting point for assessments of return on investments for funding on capacity building.

  5. encourage continuous improvement. As countries improve, the bar for performance would also get higher.

Putting the scores together creates a dashboard for each national statistical system and the ability to generate a variety of tools to enable comparison and analysis.

5 Statistical Performance Indicators

5.1 SPI Dimensions and Indicators

Each dimension of the five pillars incorporates several indicators. These Statistical Performance Indicators embody the granular measures of performance. They can be aggregated to levels of dimensions and pillars, and finally to an overall performance score to get a higher level or a more general perspective of a country’s performance.

The indicators are designed to act as proxy measures of performance for each dimension. While not comprehensive, they should add value in assessing country performance along that dimension. The intention is for the pillars and dimensions to be the focus rather than the specific indicators. The indicators for the SPI are selected following these principles: (i) use of publicly accessible data; (ii) transparent methodology; (iii) easy replicability; (iv) a time series to track performance; (v) clear portrayal of outcomes and their supporting elements; (vi) being reflective of the SDGs; (vii) enable at-a-glance comparisons on a global scale.

Benefitting from large scale data collection efforts by organizations such as the World Bank, IMF, Open Data Watch, PARIS21, the ILO, WHO, UNESCO, IHSN, and the UN, among others, 51 indicators covering 14 out of the 22 dimensions for the dashboard have been compiled. These 51 indicators provide data for each of the five pillars on data use, data services, data products, data sources, and data infrastructure. Yet, there remain major gaps in several pillars because indicators to assess performance still need to be developed, and in some cases, indicators have limited data coverage. This data availability challenge impedes efforts to measure the performance of statistical systems in certain areas and going forward countries and their international partners must work together to fill these gaps.

Below is a brief description of the indicators (or lack thereof) we have available for the 22 dimensions in the SPI framework. A detailed description of the indicators is also available in the annex. For as many as eight dimensions there was no indicator with a developed methodology, or the data collection for that measure was incomplete.

  • Dimension 1.1: Data use by national legislature: Not included because of lack of established methodology. In principle it may be possible to utilize websites of national legislatures but this will require further work and assessment.

  • Dimension 1.2: Data use by national executive branch: Not included because of lack of established methodology. There are some usable data sources (as used by (PARIS21 2019)) but gaps in data across countries have prevented full adoption.

  • Dimension 1.3: Data use by civil society: Not included because of lack of established methodology. There are some usable data sources with good coverage, for example from social media but more data is required to help assess and allow for likely biases between and within countries.

  • Dimension 1.4: Data use by academia: Not included because of lack of established methodology. We have not been able to find usable data sources with global coverage on which a new methodology could be developed.

  • Dimension 1.5: Data use by international organizations: Five measures of usefulness or reliability of country produced measures for international organizations have been included. First, on comparability of poverty estimates for the World Bank reporting on international poverty using Povcalnet ((Castaneda Aguilar et al. 2020)). Second on usable surveys for statistics on child mortality for the UN Inter-agency Group for Child Mortality Estimation ((Child Mortality Estimation (UNIGME) 2020)). Third on accuracy of debt reporting as classified by the World Bank (Source: World Bank WDI metadata). Fourth, on availability of safely managed drinking water data for use by WHO/UNICEF Joint Monitoring Programme ((Programme 2018)). Fifth, on labor force participation data for use by ILO. While these data sources provide only a partial coverag of data used by international organizations, they do provide an indication of the performance of the national statistical system.

  • Dimension 2.1: Data Releases: SDDS/e-GDDS subscription. This indicator is based on whether the country subscribes to IMF SDDS+, SDDS, or e-GDDS standards. The source is the IMF Dissemination Standards Bulletin Board. This is a reliable data source but we recognize that it is a proxy for the concept we are seeking to capture rather than a direct measurement.

  • Dimension 2.2: Online access: ODIN Open Data Openness score (Crowell et al). This is a well-established data source with good country coverage, which scores countries based on whether indicators are available online in a format that is machine readable, in a non-proprietary format, downloadable, with metadata available and terms of use. Scores range from 0-1. For more details, consult the ODIN technical documentation.

  • Dimension 2.3: Advisory/ Analytical Services: Not included because of lack of established methodology. This could be a new indicator of the number of non-recurring products on NSO website (ad hoc/experimental rather than regular releases). The indicator is the number of products found. No established source exists for this indicator.

  • Dimension 2.4: Data access services: NADA metadata. This indicator checks whether NADA microdata cataloging is available for surveys produced by NSO. NADA is an open source microdata cataloging system, compliant with the Data Documentation Initiative (DDI) and Dublin Cores RDF metadata standards. Source: NSO websites.

  • Dimension 3.1: Social Statistics: Availability of indicators for the Sustainable Development Goals 1-6, measured by an average score. The primary data source is the UN SDG database. While this is a database with comprehensive coverage that all countries have signed up to, many countries are not yet submitting all their available national data. For this reason, scores for some countries thus may not fully capture their performance in calculating the indicators. For OECD countries, we supplement the UN SDG database with comparable data submitted to the OECD following the methodology in MMeasuring Distance to the SDG Targets 2019: An Assessment of Where OECD Countries Stand.

  • Dimension 3.2: Economic Statistics: Availability of Goal 7-12 indicators, measured by an average score. See 3.1.

  • Dimension 3.3: Environmental Statistics: Availability of Goal 13 & 15 indicators, measured by an average score. Goal 14 - Life on Water - is not included because land-locked countries do not report on these indicators. See 3.1.

  • Dimension 3.4: Institutional Statistics: Availability of Goal 16-17 indicators measured by an average score. See 3.1.

  • Dimension 4.1: Censuses and Surveys: Availability of recent censuses and surveys covering broad areas. The following censuses and surveys are considered: Population & Housing census, Agriculture census, Business/establishment census, Household Survey on income/ consumption/ expenditure/ budget/ Integrated Survey, Agriculture survey, Labor Force Survey, Health/Demographic survey, Business/establishment survey. Source: NSO websites, World Bank microdata library, ILO microdata library, IHSN microdata library.

  • Dimension 4.2: Administrative Data: Availability of Civil Registration and Vital Statistics (CRVS) indicator. An ideal indicator for this dimension would include a score based on the density of administrative data available in sectors including social protection, education, labor, and health. However, social protection, education, health, and labor admin data indicators are not included because of lack of established methodology. While several promising sources for administrative data from the World Bank’s ASPIRE team, WHO, UNESCO, and ILO have been identified, these were not included due to incomplete coverage across countries. Further research and data collection effort would be needed to fill in this information, so that a more comprehensive picture of administrative data availability can be produced.

  • Dimension 4.3: Geospatial Data: Geospatial data available at 1st Admin Level. This data source from Open Data Watch focuses on data availability at the sub-national level and provides a partial understanding of a country’s ability to produce geospatial data. A research and data collection effort is needed to develop a more comprehensive global database of the availability of key geospatial indicators.

  • Dimension 4.4: Private/citizen generated data: Not included because of lack of established methodology. Currently no comprehensive source exists to measure the use of private and citizen generated data in national statistical systems, and this should be another area where more data collection is needed by the international community.

  • Dimension 5.1: Legislation and governance: This indicator is based on PARIS21 indicators on SDG 17.18.2 (national statistical legislation compliance with UN Fundamental Principles of Official Statistics), existence of National Statistical Council, national statistical strategy generation, national statistical plan. Limited country coverage makes cross country comparison limited. As a result, this is included in the dashboard, but not in the overall SPI score or index.

  • Dimension 5.2: Standards and Methods: This set of indicators is based on countries’ use of internationally accepted and recommended methodologies, classifications and standards regarding data integration. These indicators help facilitate data exchange and provide the foundation for the preparation of relevant statistical indicators. The following methods and standards are considered: System of national accounts in use, National Accounts base year, Classification of national industry, CPI base year, Classification of household consumption, Classification of status of employment, Central government accounting status, Compilation of government finance statistics, Compilation of monetary and financial statistics, Business process. Further work could improve the validity of this indicator and reduce the risk that countries may be incentivized to adopt only traditional standards and methods and neglect innovative solutions that may be more valid in the current context.

  • Dimension 5.3: Skills: Not included because of lack of established methodology or suitable data sources. A new indicator drawing on PARIS21 indicators such as statistical society presence and data literacy could be developed and is an area of future work.

  • Dimension 5.4: Partnerships: Not included because of lack of established methodology or suitable data sources. A new indicator based on textual analysis of NSS reports/websites for references to partner organizations could be developed. This is an area of future work.

  • Dimension 5.5: Finance: The indicator is based on PARIS21 SDG indicators (SDG 17.18.3 (national statistical plan that is fully funded and under implementation). It is included in dashboard, but not in the overall SPI score or index because of insufficient country coverage. For more details: (Yu Tian 2020).

5.2 Data gaps

The approach taken to the development of the SPI has been to start with a first best framework and then try and find suitable indicators to help measure progress against that framework. This has revealed a significant number of data gaps. In many critical areas of national statistical system performance, at present there is no available data to construct measures. This calls for a substantial research agenda, requiring a wide range of countries, international agencies and other organizations to work together to produce globally comparable, comprehensive, consistent and transparent information about national statistical systems.

An area of particular concern is the pillar of data use. Currently, the dashboard only features indicators for one of the five dimensions of data use, which is data use by international organizations. Indicators on whether statistical systems are providing useful data to their national governments (legislature and executive branches), to civil society, and to academia are absent. Thus the dashboard does not yet assess if national statistical systems are meeting the data needs of a large swathe of users.

Under the pillar of data services an area that needs improvement is the measurement of advisory and analytical services provided by NSOs, such as data stewardship services. By measuring this type of work done by NSOs that goes beyond producing data, the international community and the NSOs themselves can better assess whether this type of support is in place.

In the data sources pillar, more information is needed in the areas of administrative data, geospatial data, and private and citizen generated data. On administrative data, the picture is incomplete with no measures of whether countries have administrative data systems in place to measure health, education, labor, and social protection program statistics. For the geospatial indicator, there is a proxy measure of whether the country is able to produce indicators at the sub-national level, but as yet, no understanding of how countries are using geospatial information in other ways, for instance using satellite data. And while the world is increasingly awash with private and citizen generated data (e.g., on mobility, job search, or social networking), on a global scale there is no reliable source to measure how national statistical systems are incorporating this information.

Finally, several of the ‘soft’ components of the data infrastructure pillar lack adequate data. This includes the areas of skills and of partnerships between entities in the national statistical system. The dashboard makes use of the PARIS21 led SDG indicator on whether the statistical legislations in countries met the standards of the UN Fundamental Principles of Statistics, but this was not incorporated into the overall SPI score, because of inadequate country coverage. This is also true of the PARIS21 led SDG indicator on whether the national statistical system is fully funded. Countries would need to be encouraged to report on this information.

5.3 Data sources

The SPI draws on a variety of data sources to create the indicators. A guiding principle is that the SPI rely on openly available data from credible sources, such as international organizations and NSO websites. The SPI team used web scraping, accessed publicly available databases, or in some cases visited NSO websites to acquire the information. While greater detail for each specific indicator can be found in the technical documentation describing each indicator, a general overview is provided here.

For pillar 1 on data use, data is collected from four distinct sources. The World Bank supplies data for indicators on availability of comparable poverty data (from the World Bank’s Povcalnet system), and the indicator on quality of debt service data (from the World Bank’s World Development Indicators or WDI metadata). Data on the availability of under 5 child mortality data comes from the UN Inter-agency Group for Child Mortality Estimation. The indicator on availability of safely managed drinking water data is sourced from the WHO/UNICEF Joint Monitoring Programme. The indicator on the availability of source data for measuring labor force participation comes from the ILO. Each of these sources will be updated annually prior to each annual data release. The date the data is updated for each of these sources is available in our technical documentation.

For pillar 2 on data services, information on data dissemination subscription is collected from the IMF’s Dissemination Standards Bulletin Board. This and the WDI metadata follow the same update schedule and the release these two sources are identical. The online access indicator is sourced from the Open Data Watch ODIN openness score. The date of the data download is available in the technical description for this indicator. The indicator for data access services is based on (i) whether a portal is available, (ii) compliant with the Data Documentation Initiative (DDI) and with Dublin Core’s RDF metadata standards, and (iii) which has a listing of surveys and microdata sets that can provide the necessary data and reference for follow-up. This information is collected manually by visiting each NSO website.

For the data products dimension, indicators are generated using the UN Global SDG monitoring database. For each SDG indicator, the database is checked to see whether a value is available within a five year window. (for instance, for 2019 if a value is available between 2015-2019. For OECD countries, the UN SDG database is supplemented with comparable data submitted to the OECD following the methodology in Measuring Distance to the SDG Targets 2019: An Assessment of Where OECD Countries Stand. The decision to supplement the UN Global SDG monitoring database using this OECD database was taken, because a clear methodology had been established to do so. The UN Global SDG monitoring database was chosen as a primary source, rather than individual NSO websites, because data submitted to the UN Global SDG monitoring database goes through a standardized process including quality control and detailed documentation.

For dimension 4, on censuses and surveys, two complementary approaches are taken to collecting data. The first is to make use of data submitted to the World Bank, IHSN, ILO, and FAO microdata libraries. Only surveys that are marked as nationally representative are used, and surveys are classified (as either health surveys, agriculture surveys, labor force surveys, etc.) using the classifications submitted to the microdata libraries. The contents of searches on these databases are available in the github repository. The second approach is a manual data collection effort, where NSO websites have been visited to be sure no surveys were missed. To be included in this search, the survey or census must be publicly available and accessible. If surveys or censuses are missed in this search, the easiest way for a country to get it included would be to create an entry for the survey at one of the microdata libraries. Information on the completeness of the Civil Registry and Vital Statistics (CRVS) is sourced from the UN SDG global monitoring database. Information on whether data is available at the 1st administrative level, for the geospatial indicator, is sourced from Open Data Watch, as is the data openness score.

For dimension 5, data on the legislation indicator and finance indicators (compiled by PARIS21) are pulled from the UN SDG global monitoring database. Indicators in the standards and methods pillar are sourced primarily through the IMF. Information on the system of national accounts in use and national accounts base year are sourced through the World Bank’s WDI metadata. Data for the business process indicator is sourced through the United Nations Industrial Development Organization (UNIDO) and the United Nations Economic Commission for Europe (UNECE). A full description of each specific indicator is available in Appendix A6 and in the SPI github repository.

A full description of each specific indicator is available in Appendix A6 and in the SPI github repository.

6 SPI Index Methodology

An overall score is produced by combining the Statistical Performance Indicators to yield one single index. The statistical performance indicators have a nested structure, and the SPI overall score is formed by sequentially aggregating each level.

To begin we produce a score for each dimension, which, unless otherwise stated, is an unweighted average of the indicators within that dimension. For instance, the Standards and Methods dimension will be formed by taking the unweighted average of the indicators for the system of national accounts in use, national accounts base year, classification of national industry, CPI base year, classification of household consumption, etc.

\[ SPI.DIM_{ctpd} = \sum_{i=1}^{N_I} \frac{SPI.IND_{ctpdi}}{N_I} \]

where \(SPI.DIM_{ctpd}\) is pillar p, in dimension d, in time period t, and country c. \(SPI.IND_{ctpdi}\) is an indicator (e.g. population census score).

After computing a score for each dimension, a score for each pillar is computed, as either an unweighted or weighted average of the dimensions in that pillar. For pillars 1, 2, 4, and 5, the unweighted average of the dimensions within each pillar is taken. For pillar 3 on data products, we take a weighted average of the dimension scores, where the weights are based on the number of SDGs in each dimension (6 SDGs in dimension 3.1 on social statistics, 6 SDGs in dimension 3.2 on economic statistics, 2 in dimension 3.3 on environmental statistics, and 2 in dimension 3.4 on institutional statistics).2 This reflects a perspective that all SDGs are of equal importance, and therefore the dimensions are weighted accordingly. Additionally, for Pillar 4 on data sources, censuses and surveys are given separate weights, so that censuses, surveys, admin data, and geospatial data each receives a weight of 1/4. While censuses and surveys are in the same pillar in the framework, and therefore each would typically only receive a weight of 1/6 (for a total weight of 1/3) in this dimension, because of their importance in producing many indicators, they are given extra weight such that they each gets a weight of 1/4 (for a total weight of 1/2).

\[ SPI.PIL_{ctp} = \sum_{d=1}^{N_d} \frac{\omega_{pd} \times SPI.DIM_{ctpd}}{N_d} \]

\(\omega_{pd}\) is the weight for dimension d in pillar p.

After calculating the scores for each pillar, the SPI overall score is derived by taking the simple average across the 5 pillars.

The SPI overall score has a maximum score of 100 and a minimum of 0. A score of 100 would indicate that a country has every single element that we measure. A score of 0 indicates that none are in available. To be precise:

\[ SPI.INDEX_{ct} = \sum_{p=1}^{N_p} \frac{SPI.PIL_{ctp}}{N_p} \]

Where SPI.INDEX is the SPI overall score. SPI.PIL are the 5 SPI pillars listed above. In the notation, c is a country, t is the date, p is a pillar

The nested structure of our index and the summation methods used to build an overall score ensure the axiomatic properties outlined in (Grant J. Cameron et al., n.d.).3 These properties include symmetry, monotonicity, and subgroup decomposability. Symmetry refers to property where if the values of two indicators in a nesting are switched, then the resulting index scores are unaffected. Monotonicity implies that if the value of an indicator improves, then the resulting index scores improve as well. Subgroup decomposability results from the fact that the scores are a weighted average of the subgroups (either indicators, dimensions, pillars) that make up that score and so can be written as a linear combination of those subgroups.

7 SPI Overall Scores

The purpose of the SPI is to help countries assess and improve the performance of their statistical systems. The presentation of SPI overall scores is designed to reflect that aim. Small differences between countries should not be highlighted since they can reflect imprecision arising from the currently available indicators rather than meaningful differences in performance. Instead, the presentation of overall SPI scores focuses on larger groupings of countries reflecting broad categories of performance as measured by the indicator framework. In total, there are 174 countries with sufficient data to compute an index value. This set of countries covers 99.2 percent of the world population.

The map is color coded based on the performance of countries on our index. Given the imprecision inherent in the calculations we recommend that the color coding provides the most detailed subdivisions of maturity. Finer distinctions are unlikely to provide meaningful differentiation between countries.

Countries are grouped into five categories as shown on the map in figure 6.1:

Figure 6.1: SPI Overall Score by Quintile in 2019

The countries scoring in the top 20% have an average SPI overall score of 86.4. A maximum score of 100 would indicate that a country has every single element in place that is measured through the SPI. Meanwhile, countries in the bottom 20% score significantly worse, with average scores of 37.5. In Pillar 1, countries in the top quintile have a score of 98.9, while countries in the bottom quintile have scores of 53. One area where even top quintile countries are not doing as well as possible is on our data products pillar. Countries in the top quintile score only 72.3 points out of 100 in this area. Pillar 3 measures whether countries have a value in the past 5 years for each SDG indicator, meaning even the top quintile countries have only around 72.3% of indicators averaging over the SDGs.

Table 6.1: Table of SPI Overall and Pillar Scores by Quintile Group in 2019

To highlight a few specific indicators, countries in the top 20% on average get nearly perfect scores in terms of the debt reporting to the World Bank. The top 20% of countries in the SPI overall score have an average score on their debt reporting of 1, with a maximum score of 1 and minimum score of 0, while those in the bottom 20% on the SPI overall score have an average score of 0.7. To take another example, when looking at scores on whether a population census has been conducted recently, countries in the top 20% score on average 1 on our population census indicator, with a max score of 1 and minimum score of 0, while those in the bottom 20% on our SPI overall score receive 0.8 points on the population census indicator. The table below shows differences across maturity groups for a select set of indicators.

Table 6.2: Table of SPI Overall and Select SPI Indicators by Quintile Group in 2019

7.1 SPI Overall Score by Region and Income Group

There are large differences in the SPI overall score across World Bank regions and income groups. Overall, North America has the highest average overall SPI score, while the Sub-Saharan Africa region has the lowest average score.4 There is also a clear gradient with respect to income groups. Countries classified as low income have lower scores on average than countries classified as middle income. High income countries have the highest average SPI overall score.

Finally, when looking over time, since 2016 the SPI overall score values are relatively stable across regions. There has been some modest increase in the SPI overall score scores for middle income and high income countries, with little progress for low income countries.

Figure 6.2.A: SPI Overall Score by Region in 2019

Figure 6.2.B: SPI Overall Score by Income Group in 2019

Figure 6.2.C: SPI Overall Score by Lending Group in 2019

Figure 6.2.D: SPI Overall Score by Fragile/Conflict Situation Status in 2019

Figure 6.2.E: SPI Overall Score by Year (2016-19)

As well as large differences across regions, there is significant variation in the SPI overall scores within regions. For instance, in the Latin America & Caribbean World Bank region, Mexico is the highest scoring country with a score of 87.5. However, Haiti, the lowest scoring country in the region, earns a substantially lower score of 37.5. In Sub-Saharan Africa, the highest scoring country is Mauritius with a score of 75.9, while the lowest scoring country is Somalia with a score of 19.6. In the East Asia and Pacific region, the top scoring country is Korea, Rep. with a score of 88.3, while the lowest scoring country is Marshall Islands with a score of 20.9.

Figure 6.3: SPI Overall Scores by Country and Region in 2019

7.2 SPI Scores by Pillar

By presenting scores for each of the pillars – data use, data services, data products, data sources, and data infrastructure – that compose the SPI overall score, it is possible to show where variation across countries is coming from and to assess the specific areas in which countries may be struggling

7.2.1 Pillar 1: Data Use

Figure 6.4 shows a world map displaying countries based on data use score quintiles Overall, the high income countries in North America and Europe tend to rate most highly along this dimension, whereas Sub Saharan African countries tend to lag.

Figure 6.4: Pillar 1: Data Use Scores by Quintile in 2019

Countries within region show significant dispersion in performance . It is worth noting that several countries such as the United States, Mexico, Finland, and Costa Rica, among others, receive the maximum possible score in this dimension of 100. Several countries score the minimum score as well, including the Syrian Arab Republic, South Sudan, Namibia, and Nauru, among others.

Figure 6.5: Pillar 1: Data Use Scores by Country and Region in 2019

The table below shows the average score on each indicator in pillar 1. Countries in the top 20% possess nearly all of the components in pillar 1, while countries in the bottom 20% are lacking in many areas. Countries in the bottom 20% have an average of 0.1 on the comparable poverty estimate indicator, an average of 0.6 on the availability of under 5 mortality data, and an average of 0.7 on debt reporting.

Table 6.3: Select Pillar 1 Indicator Scores by SPI Overall Score Quintile Group in 2019

7.2.2 Pillar 2: Data Services

For pillar 2 on data services, high income countries tend to score highest along this pillar India, Morocco, and the Philippines scored highly as well in this pillar. This pillar contains four indicators on the quality of data releases, online accessibility, advisory and analytical services (not included due to lack of data), and the availability and use of data access services.

Figure 6.6: Pillar 2: Data Services Scores by Quintile in 2019

Figure 6.7: Pillar 2: Data Services Scores by Country and Region in 2019

In terms of specific indicators, the top 20% of countries receive an average score of 1 on the indicator measuring country adoption of the IMF’s data dissemination standards, while those in the bottom 20% receive an average score of 0.5. Countries in the bottom 20% receive an average data openness score, produced by Open Data Watch, of 0. This indicator is made up of several sub-indicators including whether data is available in machine readable format, a non-proprietary format, has download options, metadata available, and terms of use. For instance, in the bottom 20% of countries the score on whether data is available in a machine readable format is only 0.3 out of a maximum of 1. Additionally, countries in the bottom 20% only score an average of 0.4 on the measure of whether metadata meeting the standards of the Database Documentation Initiative (DDI) is available on surveys conducted.5

Table 6.4: Select Pillar 2 Indicator Scores by SPI Overall Score Quintile Group in 2019

7.2.3 Pillar 3: Data Products

The data products pillar is based on whether countries are reporting data to monitor the SDGs. The SDG goals and targets provide a comprehensive, internationally agreed framework, encompassing indicators reflecting all domains of official statistics: economic, social, environmental and institutional which is relevant to all nations. For this pillar, indicators are produced using the UN Global SDG monitoring database. For each SDG indicator, a value is checked for availability within a five year window of a particular year. For instance, for 2019, is any indicator value available between the years 2015-2019.

For this indicator, there is a weaker relationship between a country’s income level and their performance in this pillar. No countries receive a maximum score of 100, which would indicate that they report on every SDG indicator at least once inside a 5 year window. Additionally, no country scores 0, which would indicate that they have reported no information for any of the SDG indicators.

The results for this pillar reflect two important aspects of the performance of a national statistical system, both of which need to be in place for a good score. First, do the SDG indicators required by the globally agreed framework exist for that country? And second, are those indicators available, in the agreed formats, on the UN Global SDG monitoring database? It is likely that some countries are producing more indicators but are not making them available through an internationally comparable database. For OECD countries, we supplemented the UN SDG database with comparable data submitted to the OECD following the methodology in (OECD 2019). The UN Global SDG monitoring database has been supplemented using this OECD database, because a clear methodology had been established to do so. Even with this supplemental data from the OECD included, there is considerable room for improvement in reporting on the SDGs.

Indicator values that are either country reported, country adjusted, estimated, or is included as global monitoring data are included. Values that were produced by an international organization through modeling are excluded. These classifications are based on the UN SDG metadata, where for each value of the indicator, the responsible international agency has been requested to indicate whether the national data were adjusted, estimated, modeled or are the result of a harmonized global monitoring exercise. The “nature” of the data classification in the SDG database is determined as follows:

  • Country data (C): Produced and disseminated by the country (including data adjusted by the country to meet international standards);

  • Country data adjusted (CA): Produced and provided by the country, but adjusted by the international agency for international comparability to comply with internationally agreed standards, definitions and classifications;

  • Estimated (E): Estimated based on national data, such as surveys or administrative records, or other sources but on the same variable being estimated, produced by the international agency when country data for some year(s) is not available, when multiple sources exist, or when there are data quality issues;

  • Modeled (M): Modeled by the agency on the basis of other covariates when there is a complete lack of data on the variable being estimated;

  • Global monitoring data (G): Produced on a regular basis by the designated agency for global monitoring, based on country data. There is no corresponding figure at the country level.

Figure 6.8: Pillar 3: Data Products Scores by Quintile in 2019

Figure 6.9: Pillar 3: Data Products Scores by Country and Region in 2019

Even countries in the top quintile have significant room for improvement in reporting on the SDGs. The average score for the top 20% of countries is 72.3, while for the bottom 20% it is 48. For a specific goal, such as for SDG 1 on poverty, the top 20% receives an average of 0.8 out of a maximum of 1. This means that the top performing countries are only reporting on 80 percent of the SDG indicators between 2015-19. The bottom 20% receive an average score of 0.4. The table below shows the full set of breakdowns by SDGs, suggesting reporting on SDG 5 on Gender lags other goals.

Table 6.5: Select Pillar 3 Indicator Scores by SPI Overall Score Quintile Group in 2019

7.2.4 Pillar 4: Data Sources

Pillar 4 examines whether countries have the data sources available that are necessary to produce statistics for public use. It includes three aspects of data sources: censuses and surveys, administrative data, and geospatial data. Private and citizen generated data is an important area that is at present not incorporated due to the lack of an established source. For censuses and surveys the score reflects both whether a data source exists and how recently the source was produced, as both are needed for accurate reporting of current conditions. For administrative data and geospatial data, proxies regarding the state of these data systems are used.

Figure 6.10: Pillar 4: Data Sources Scores by Quintile in 2019

When looking at the country chart, again there is significant variability across countries. No country receives the maximum score of 100, with many countries falling short particularly on our geospatial indicator. Somalia is the only country that receives a score of 0.

Figure 6.11: Pillar 4: Data Sources Scores by Country and Region in 2019

Countries in the top 20% have an average score of 72.5, while countries in the bottom 20% have an average score of 22.3. The gap between the top 20% and bottom 20% for the population and housing census indicator is 0.2 points on a scale from 0 to 1. The gap for countries conducting a business/establishment survey is 0.9. The scores on the geospatial indicator are low. The average score for the top 20% is 0.3, while it is 0.1 for the bottom 20%.

Table 6.6: Select Pillar 4 Indicator Scores by SPI Overall Score Quintile Group in 2019

7.2.5 Pillar 5: Data Infrastructure

The fifth pillar measures whether countries have the hard and soft infrastructure to produce the data sources, data products, and data services to produce useful data. For the pillar 5 sub-score, the only usable data is on a set of ten methods, standards, and classifications. Internationally accepted and recommended methodology, classifications and standards provide the basis for national statistical offices (NSOs) on data integration, facilitating data exchange and providing the foundation for the preparation of relevant statistical indicators. The following methods and standards are considered: System of national accounts in use, National Accounts base year, Classification of national industry, CPI base year, Classification of household consumption, Classification of status of employment, Central government accounting status, Compilation of government finance statistics, Compilation of monetary and financial statistics, and Business process. Data has also been collected on statistical legislation and governance, as well as on finance, but these indicators currently lack adequate country coverage to include in the index.

Countries scoring in the top 20% tend to be concentrated among the high income countries. These top scoring countries have an average score for pillar 5 of 97.1, which is near the maximum score of 100. Countries in the bottom 20% have an average score fo 25.4.

Figure 6.12: Pillar 5: Data Infrastructure Scores by Quintile in 2019

Several countries score the maximum of 100 in the data infrastructure pillar One country, Somalia, scores 0 points in this pillar.

Figure 6.13: Pillar 5: Data Infrastructure Scores by Country and Region in 2019

Among countries in the top 20%, the average score is near 100 across all the indicators of pillar 5. In the bottom 20%, countries on average score close to zero points for the CPI base year, classification of status of employment, central government accounting status, compilation of government finance statistics, and business process indicators. Bottom 20% countries, score above 0.5 points on the system of national accounts in use and compilation of monetary and financial statistics indicators.

Table 6.8: Select Pillar 5 Indicator Scores by SPI Overall Score Quintile Group in 2019

7.2.6 Correlations between SPI pillars

In the following chart, correlations between the SPI overall score and the individual SPI pillar scores are shown. All pillars are positively correlated with one another. At the same time, no dimension is perfectly correlated with any of the other dimensions, which would indicate that a dimension was not providing any additional information on the statistical performance of countries. The pillar with the single highest correlation with the overall measure is pillar 1 on data use; pillar 3 on data products has the lowest overall correlation.

Figure 6.14: Correlation Between SPI pillars in 2019

7.3 SPI Scores by Dimension and Indicator

While there are large differences across regions, income groups, and countries in the SPI overall score, there are also differences across indicators in the strengths and weaknesses of countries that have the same final score. For instance, some countries may reach a final score by excelling in the area of data sources, while others may reach a final score by excelling in data infrastructure. There are many paths that countries may take to reach an SPI overall score and only by studying the pillars and indicators for a particular country in detail can you understand why a country gets a particular score. To help interpret the scores, it is possible to analyze which of the pillars and indicators are most responsible for differences between countries.

In order to do so, a leave-out approach to calculating the SPI overall scores has been taken, which can help to understand which pillars and indicators are most impactful. The leave-out approach is similar in spirit to a jackknife approach (see (Miller 1974) for an introduction), which has been used in a variety of situations to examine the sensitivity of estimates to leaving out specific observations. Specifically for this case, the leave-out approach consists of sequentially deleting indicators one at a time and recalculating the SPI overall scores. By sequentially omitting indicators, the total difference between the SPI overall score incorporating all indicators and an alternative SPI score that is calculated by omitting a single indicator can be computed.The pillars and indicators can then be ranked, based on which produce the greatest total difference between the SPI overall score and the alternative score.

Specifically, the approach is as follows:

  1. Calculate the SPI overall score for each country using all dimensions and indicators, \(SPI.INDEX_{c}\).

  2. For each indicator \(j=1,...,J\) do:

  • Generate an alternative SPI score omitting indicator \(j\), \(SPI.INDEX^{-j}_{c}\).

  • Calculate difference between the original score and the alternative score for each country in a year \(t\), \(e_{c}^{-j}=SPI.INDEX_{c}-SPI.INDEX^{-j}_{c}\)

  • Calculate the mean absolute difference across all countries, \(E^{-j}=\sum_{c=1}^{N_c} |e_{c}^{-j}|/C\)

  1. Sort indicators based on \(E^{-j}\).

Figure 6.15: SPI Indicator Importance for Top 10 and Bottom 10 Indicators in 2019

The indicator that drives the single largest difference between the SPI overall score and an alternative score omitting the indicator is the NADA metadata availability indicator in dimension 2.4 on data access services. This is closely followed by the indicator on whether geospatial information is available and whether a complete civil registration and vital statistics system (CRVS) is available. The indicators creating the smallest difference when omitted tend to the be the individual indicators in pillar 3 on data products. The indicator with the single smallest difference is in Pillar 3:4 on SDG 4

8 Analysis

8.1 Unique Values

As a check of the data, the number of unique scores for the SPI overall score is calculated. If the SPI overall score produces a large number of tied scores, for instance, then the index will be less able to distinguish between the statistical performance of countries. When calculating the number of unique values for 2019, it is found that there are 174 unique scores for 174 countries. This means there are 0 tied values.

When looking at each specific pillar, there are only 18 unique scores for pillar on data use. The data use indicator is coming solely from dimension 1.5 on data use by international organizations. For pillar, there are 163 unique scores. For pillar 3, there are 173 unique scores. There are 173 unique scores for pillar 4, and there are 20 unique scores for pillar 5.

8.2 Relationship to GDP Per Capita and the Human Capital Index

The correlation between the SPI overall score and and the log of GDP per capita and the World Bank’s Human Capital Index ((Bank 2020)) provide a face validity check between the SPI index and other outcomes. This analysis is not meant to assert a causal relationship, only to assess whether the SPI index is correlated with other outcomes in ways that might be expected. The source for GDP per capita comes from the World Bank’s World Development Indicators (WDI) database (NY.GDP.PCAP.KD). The GDP per capita numbers are in constant 2010 US$.

It would be expected that a strong positive relationship between GDP per capita of countries and their statistical system, as higher income countries would tend to have more resources available for statistical production. In fact, there is a strong relationship between the two. The correlation in 2019 between log GDP per capita and the SPI overall score is 0.66.6

Another measure of a country’s development is the Human Capital Index (HCI) developed by the World Bank ((Kraay 2019)). The Human Capital Index is designed to capture the amount of human capital a child born today can expect to attain by age 18 in a country. The index combines a country’s child mortality, learning adjusted years of schooling, adult survival rates & stunting into one index.7. Again, a strong positive relationship between a country’s HCI value and their Statistical Performance Indicators index might be expected, as countries with a more developed human capital stock are likely to have greater capacity to produce statistics. Again, this is what is seen. The correlation between the 2018 value of the HCI (the latest value available at the time of this writing) and the 2018 value of the SPI overall score is 0.79.

The scatter plot of the relationship between log GDP per capita, the HCI, and our SPI overall score for the years 2016-2019 is shown below. In general, countries with higher per capita income and higher levels of human capital tend to have better performing statistical systems according to the SPI measure.

Figure 7.1: Plot of SPI overall score on Human Capital Index and GDP per capita

So as to highlight countries where this relationships do not hold as well, the next figure shows the 15 countries that most over-perform and the 15 countries that most under-perform on the SPI Index compared to their levels of GDP per capita and the Human Capital Index. A perfect fit between the SPI overall score and GDP per capita and the Human Capital Index is not to be expected, as countries differ in the resources put into their statistical system, even conditional on their levels of development. Highlighting outliers can sometimes be a useful exercise for determining whether a measure is identifying on the ground realities.

In order to produce this figure, an OLS regression of the SPI overall score in 2019 on log GDP per capita has been estimated.The residual, which can be interpreted as the difference between the country’s SPI overall score value and the expected index value based on their GDP per capita has then been calculated. Countries with values of the residual greater than zero are over-performing based on their GDP per capita and countries with residuals less than zero are under-performing. The corresponding figure for the Human Capital Index is calculated similarly.

This figure identifies some countries that appear to have better performing statistical systems than might be expected (Rwanda, Uganda, Egypt, Mexico, Philippines, Armenia and Turkey appear in green on both charts). There are also countries that appear to have poorer performing statistical systems than expected: there are several small island states for example that appear red on both charts.

Figure 7.2: Top 15 Over/Under-Performers on SPI overall score compared to GDP per capita and Human Capital Index in 2019

8.3 Relationship to Government Effectiveness

A common justification for improving statistical systems is that doing so can lead to better governance. Without good statistics, countries may be flying blind on where to target resources to improve the public welfare. Also, good statistics can help hold public officials accountable for progress toward reaching a country’s goals. In this next section, it is shown that the relationship between our SPI overall score and an estimate of government effectiveness produced by the Worldwide Governance Indicators (WGI) is analyzed. A strong relationship between the SPI measure of statistical performance and the WGI measure of governmental effectiveness is found.

(Kraay, Kaufmann, and Mastruzzi 2010) produce a set of Worldwide Governance Indicators, including a measure of government effectiveness. According to the WGI metadata, the government effectiveness indicator captures perceptions of the quality of public services, the quality of the civil service and the degree of its independence from political pressures, the quality of policy formulation and implementation, and the credibility of the government’s commitment to such policies. The estimate gives the country’s score on the aggregate indicator, in units of a standard normal distribution, i.e. ranging from approximately -2.5 to 2.5.8. The government effectiveness indicator is available from 1996 to 2019.

There is a strong relationship between the SPI overall score and the government effectiveness indicator. The correlation between the two in 2019 is 0.77. The scatterplot below shows the relationship between the SPI overall score and the government effectiveness indicator.

Figure 7.3: Plot of SPI overall score on Government Effectiveness in 2019

To tease out to what extent the relationship between the government effectiveness indicator and the SPI is due to other factors such as income or regional characteristics, the results from an OLS regression are presented below. While it is acknowledged that a more detailed study could be conducted to better understand the processes relating the two, a relationship is found between government effectiveness and statistical performance after accounting for income and regional characteristics of a country.

The regression model used takes the following form:

\[ G_{ctr} = \alpha_t + \gamma_r + \beta SPI.INDEX_{ctr} + \theta X_{ctr} + \epsilon_{ctr} \]

where \(Y_{ctri}\) is the government effectiveness estimate for country i, in time period t, and region r. \(SPI.INDEX_{ctri}\) is the SPI overall score. \(X_{ctri}\) is a set of control variables in the regression. This includes log GDP per capita from the World Bank WDI. \(\epsilon_{ctri}\) is the error term. \(\alpha_t\) is an indicator variable for each year and \(\gamma_r\) is a regional indicator variable.

The table below shows the estimate of \(\beta\), the effect of the statistical performance measure on government effectiveness. Full results from this regression are shown in a table in the appendix. The estimated coefficient is statistically significant at the 0.1% level, and implies that a 10 point increase in the SPI overall score is associated with a 0.2 standard deviation increase in government effectiveness. For context, this 10 point jump in the SPI would roughly take a country from roughly the median in terms of government effectiveness to roughly the 58th percentile.

Table 7.1: Linear Regressions of Government Effectiveness Score on SPI Overall Score from 2016-19

8.4 Country Changes in the Index

In order to assess how stable the index values are over time, comparisons have been made between the index values in 2016 and the 2019 values. Overall, the SPI overall score is quite stable over time. The correlation between the 2016 value and the 2019 value is 0.96.

Figure 7.4: Scatterplot of 2019 SPI overall score & 2016 SPI overall score

While the scores were relatively stable over time, some countries did see large improvements in their score from 2016-2019. The country that improved most on the index from 2016 to 2019 was Myanmar. Myanmar improved by 20.3 points out of 100. The table below shows the changes in the SPI overall score for the top 10 largest improvers.

Table 7.2: Top 10 Countries with Largest Changes from 2016-2019.

8.5 Density Plots

As another check, the distribution of scores across countries for each year has been presented and compared to a normal distribution. This exercise checks for whether the distribution of the SPI overall scores contains significant skew or fat tails. There is some indication of a bunching of scores near the top of the distribution of SPI scores. This is due to a large number of OECD countries possessing similar scores. This is not unexpected as OECD requires member countries to adhere to several methodological standards and to regularly report on a large set of indicators. These countries also are composed of several of the highest income countries that tend to be on the frontier of statistical production.

Figure 7.5: Distribution of SPI overall score across Countries

9 Further Project Information

9.1 Process for Inquiring about and Validating the Data

The data for the Statistical Performance Indicators are collected from established public and open sources. The team makes every effort to ensure that the data presented in the Statistical Performance Indicators are accurate, but it is possible that the sources used to assign values for the indicators are occasionally not up to date or accurate despite these efforts. Countries and all other users have the opportunity to inquire about the values that make up the indicators through contacting the Bank directly or via .

9.2 Process for Updating Indicators

While the framework put forth in this note is designed to capture the contours of statistical systems around the world over the next decade and beyond, the indicators themselves are expected to improve over time. Changing indicators over time does come with a tradeoff: while it would improve the measurement of statistical performance, it would break the comparability with previous measures. Recognizing this trade-off, we plan to follow a 2- to 3-year cycle when, new indicators may be introduced and current indicators re-evaluated based on feedback from stakeholders and users, including national authorities. Whenever such a change takes place, the historical SPI series would need to be updated for comparability over time. In order to be completely transparent, all changes to methodology will be tracked through a publicly available github repository and all code and underlying data to produce the indicators will be published.

10 Conclusion

The new Statistical Performance Indicators (SPI) will replace the Statistical Capacity Index (SCI), which the World Bank has regularly published since 2004. Although the goals are the same, to offer a better tool to measure the statistical systems of countries, the new SPI framework has expanded into new areas including in the areas of data use, administrative data, geospatial data, data services, and data infrastructure. The SPI provides a framework that can help countries measure where they stand in several dimensions and offers an ambitious measurement agenda for the international community.

The goal of the SPI is to offer a framework that is forward looking, measures less mature statistical systems as well as advanced systems, covers the broader national statistical system beyond the National Statistical Office (NSO), and gives countries incentives to build a modern statistical system. The project uses open data and open code to build confidence in the work. The data will also be updated on a yearly basis to track progress over time.

More research and data collection is, however, needed to improve measurement. Several of the dimensions of the SPI do not have measurable indicators yet. One of the functions of the dashboard is to motivate action on the part of the international community to help improve the collection of data so we can better measure these areas.

Countries and international organizations need to know the current capabilities of national statistical systems, and the new SPI is meant to provide insights into this. Through the SPI, there is the potential for countries and donors to create mechanisms for learning from their peers to develop a virtuous cycle of investment, effectiveness, innovation, value added, and impact.

11 Appendix

11.1 Country SPI overall scores

Below, the full list of countries by their SPI overall score in 2019 is presented. The first column is the country name and the following columns are the overall SPI overall score, and then the sub-scores for pillars 1,2,3,4 and 5.

The purpose of the SPI is to help countries assess and improve the performance of their statistical systems. The presentation of SPI overall scores is designed to reflect that aim. Small differences between countries should not be highlighted since they are likely to reflect imprecision arising from the methodology rather than meaningful differences in performance. Instead, presentation of overall SPI scores focuses on larger groupings of countries reflecting broad categories of performance as measured by the indicator framework.

Countries shaded in dark orange are the lowest performing, countries in dark green are the highest performing. Countries are grouped into five groups:

  • Top Quintile: Countries in the Top quintile are classified in this group. Shading in dark green.
  • 4th Quintile: Countries in the 4th quintile, or those above the 60th percentile but below the 80th percentile are in this group. Shading in light green.
  • 3rd Quintile: Countries in the 3rd quintile, or those between the 40th and 60th percentile, are classified in this group. Shading in yellow.
  • 2nd Quintile: Countries in the 2nd quintile, or those above the 20th percentile but below the 40th percentile, are in this group. Shading in light orange.
  • Bottom 20%: Countries in the bottom 20% are classified in this group. Shading in dark orange .

Table A.1: SPI overall score in 2019 and Pillar Scores

11.2 Country SPI overall scores over time

Table A.2: SPI overall scores over time

11.3 SPI Overall and Pillar Scores by Region, Income, & Lending

Table A.3: Table of SPI Overall and Pillar Scores by Region in 2019

Table A.4: Table of SPI Overall and Pillar Scores by Income Group in 2019

Table A.5: Table of SPI Overall and Pillar Scores by Lending Group in 2019

Figure A.1: SPI Pillar 1 - Data Use Score - By Region in 2019

Figure A.2: SPI Pillar 2 - Data Services Score - By Region in 2019

Figure A.3: SPI Pillar 3 - Data Products Score - By Region in 2019

Figure A.4: SPI Pillar 4 - Data Sources Score - By Region in 2019

Figure A.5: SPI Pillar 5 - Data Infrastructure Score - By Region in 2019

11.4 Comparison to other measures of statistical performance

Next, the SPI is compared with several other indices of statistical performance that have been created. This provides a sense of how rankings differ across measures, how they correlate with other outcomes, and how the distributions of scores compare. These are the SCI, the Open Data Watch index (ODIN), and version 0 of the SPI overall score that was produced in (Grant J. Cameron et al. 2019)

We first compare the SPI overall score to the older World Bank Statistical Capacity Index. The correlation between the SPI overall score and the SCI is 0.765.

Figure A.6: Scatterplot of Statistical Capacity Index (SCI) and SPI Overall Score

Next, the relationship is shown between Log GDP per capita, the SCI and the SPI overall score using linear regression. The Open Data Watch ODIN score and the version of the Statistical Performance Index developed in (Grant J. Cameron et al. 2019) are also included. While showing a strong relationship between an index and log GDP per capita does not mean the index is necessarily correct, and is certainly not necessarily causal, it does provide a face validity check of the index. Heteroskedasticity robust standard errors are shown in the table.

Overall, the new SPI overall score has the strongest relationship to GDP per capita. The linear regression estimates indicate that a 1% increase in GDP per capita is associated with a 0.1 point increase in the SPI overall score. The r-squared from this regression is 0.38.

Table A.6: Relationship between Statical Performance Measures and GDP per capita.

We compare the SPI overall score to the Open Data Watch rankings of country statistical systems. The correlation between the SPI overall score and the ODIN index is 0.84

As a final comparison, the new SPI overall score in compared with thehe index developed by (Grant J. Cameron et al. 2019), which can be thought of as a version 0 of t SPI overall score. The authors use similar data sources for their index. However, there are some differences.

First, the similarities. The methodology for constructing the index is the same. Also, the censuses and surveys (Indicator 4.1) and standards and methods (Indicator 5.2) are identical. The indicator for data releases (Indicator 2.1) and data services (Indicator 2.4) are pulled from the information collected in the fourth dimension on dissemination practices from version 0 of the SPI in Cameron et al. (2019). Finally, both indicators include an indicator for Complete Vital Registration Statistics (CRVS). The CRVS indicator is in the administrative data section of the new SPI overall score, while it was in the standards, methods, and classifications section of the SPI version 0.

For the differences, SPI version 0 had four dimensions, namely: (i) Methodology, Standards and Classifications (MSC), which provides information on the technology being used by the NSS; (ii) Census and Surveys (CS), which describes the intermediate products of the NSS; (iii) Availability of Key Indicators (AKI), which focuses on key final products needed for policy; and (iv) Dissemination Practices and Openness (DPO), which evaluates the extent to which products are publicly disseminated.

The indicator on AKI, from version 0, is conceptually similar to the Data Products dimension in the new SPI, but uses different sources of data and the DPO section is similar to the SPI data services section, but draws on some different sources in some cases.

We compare the SPI overall score to the SPI version 0. The correlation between the SPI overall score and the version 0 index is 0.932

Figure A.7: Scatterplot of new SPI overall score on SPI Version 0 Index

11.5 Other Tables and Figures

Table A.7: Linear Regressions of Government Effectiveness Score on SPI Overall Score from 2016-19

11.6 Indicator Metadata

Table A.8: SPI Indicator Metadata

References

Cameron, Grant J., Hai-Anh H. Dang, Mustafa Dinc, James Foster, and Michael M. Lokshin. n.d. “Measuring the Statistical Capacity of Nations.” Oxford Bulletin of Economics and Statistics n/a (n/a). https://doi.org/https://doi.org/10.1111/obes.12421.
Cameron, Grant J, Hai-Anh H Dang, Mustafa Dinc, James Foster, and Michael M Lokshin. 2019. “Measuring the Statistical Capacity of Nations.”
Castaneda Aguilar, R Andres, Tony Fujs, Dean Jolliffe, Christoph Lakner, Daniel Gerszon Mahler, Minh C Nguyen, Marta Schoch, et al. 2020. “September 2020 PovcalNet Update: What’s New.” World Bank.
Child Mortality Estimation (UNIGME), United Nations Inter-Agency Group for. 2020. “Levels & Trends in Child Mortality: Report 2020—Estimates Developed by the United Nations Inter-Agency Group for Child Mortality.” United Nations Children’s Fund.
Kraay, Aart. 2019. “The World Bank Human Capital Index: A Guide.” The World Bank Research Observer 34 (1): 1–33. https://doi.org/10.1093/wbro/lkz001.
Kraay, Aart, Daniel Kaufmann, and Massimo Mastruzzi. 2010. The Worldwide Governance Indicators: Methodology and Analytical Issues. The World Bank.
Miller, Rupert G. 1974. “The Jackknife-a Review.” Biometrika 61 (1): 1–15.
OECD. 2019. Measuring Distance to the SDG Targets 2019. https://doi.org/https://doi.org/https://doi.org/10.1787/a8caf3fa-en.
PARIS21. 2019. “Statistical Capacity Development Outlook 2019.” http://statisticalcapacitymonitor.org/pdf/Statistical.
Programme, WHO/UNICEF Joint Monitoring. 2018. “JMP Methodology: 2017 Update & SDG Baselines.” World Health Organization; United Nations Children’s Fund (UNICEF) Geneva ….
Yu Tian. 2020. “THE PARTNER REPORT ON SUPPORT TO STATISTICS PRESS 2020.” PARIS21. https://paris21.org/sites/default/files/inline-files/PRESS2020.

  1. The code and data are available on [github]{https://github.com/worldbank/SPI} licensed under the [Creative Commons Attribution 4.0 International License]{https://creativecommons.org/licenses/by/4.0/}, which allows users to freely share, copy, and redistribute any of the materials, as well as adapt, remix, transform, and build upon the material for any purpose, so long as appropriate credit is given the SPI team and the other teams described under the license.↩︎

  2. SDG 14 - Life Below Water - is omitted because land-locked countries do not report on these indicators.↩︎

  3. An earlier version of the journal publication was released as a World Bank Policy Research paper, (Grant J. Cameron et al. 2019)↩︎

  4. All tables and figures show unweighted summary statistics (i.e. the summary statistics do not weight by population).↩︎

  5. For more information, see https://ddialliance.org/↩︎

  6. To understand what effect taking the log has on this correlation, the correlation in 2019 between (non-logged) GDP per capita and the SPI overall score is 0.58.↩︎

  7. For more details, visit the Human Capital Index website↩︎

  8. Detailed documentation of the WGI, interactive tools for exploring the data, and full access to the underlying source data available at www.govindicators.org. The WGI are produced by Daniel Kaufmann (Natural Resource Governance Institute and Brookings Institution) and Aart Kraay (World Bank Development Research Group).↩︎